The semantic typology of visually grounded paraphrases

Visually grounded paraphrases (VGPs) are different phrasal expressions describing the same visual concept in an image. Previous studies treat VGP identification as a binary classification task, which ignores various phenomena behind VGPs (i.e., linguistic interpretation of concept) such and from aspects. In this paper, we propose semantic typology for VGPs, aiming to elucidate deepen understanding about how human beings interpret vision with language. We construct large dataset that annotates class each pair belongs according our typology. addition, present model fuses language features on dataset. Experiments indicate joint representation learning is important classification. further demonstrate can boost performance visually textual entailment.